1
Real-Time Machine Learning Pipeline: A Clinical
Early Warning Score (EWS) Use-Case
Session 62, February 12, 2019
Prem Timsina, Senior Data Engineer, Mount Sinai Health Systems
Arash Kia, Senior Data Scientist, Mount Sinai Health Systems
2
Prem Timsina, Sc.D.
Arash Kia, M.D., M.Sc.
There is no real or apparent conflicts of interest to report
Conflict of Interest
3
ā¢ Arash Kia: M.D., M.Sc. - Senior Data Scientist
ā¢ Prem Timsina: D.Sc. - Senior Data Engineer
ā¢ Fu-Yuan Cheng: M.Sc. - Junior Data Scientist
ā¢ Prathamesh Parchure: M.Sc. - Data Science Analyst
ā¢ William Otto: B.Sc. - System Admin, DevOps Engineer
ā¢ Robert Freeman: M.Sc., RN, NE, B.Sc. - Senior Director of Operations
ā¢ Matthew A. Levin: M.D. - Clinical Lead, Associate Professor, Department of Anesthesiology,
Division of Cardiothoracic Anesthesia
Clinical Data Science Team
4
Agenda
Problem Definition
Proposed Solution
Use Cases
Challenges
Future Plan and Vision
5
ā¢ Explain the rationale behind the need for a real-time machine
learning (ML) pipeline in personalized medicine
ā¢ Explain the architecture of the healthcare ML pipeline
ā¢ Identify the challenges and the lessons learned from developing
and deploying process
ā¢ Discuss how the ML pipeline can be generalized into various
health care settings and cases of use
Learning Objectives
6
Problem Definitions
8
Optimization Opportunity
ļ± Clinical Decision
ā¢ A process to organized clinical knowledge linked to patient
information to improve patient's health and healthcare delivery
ļ± Strategies for effective use of knowledge
ā¢ Best knowledge available in terms of adoptability and effectiveness
ā¢ Continuous improvement of knowledge
ā¢ Data-driven CDS
9
Clinical Decision Support Cycle
10
Clinical Decision Support Timeline
11
ļ± Data
ļ¼ Quality and variability of clinical data
ļ¼ Interoperability of clinical data
ļ¼ Data sharing
ļ¼ Standardization of data representation both terminology and ontology level
ļ± Knowledge
ļ¼ Lack of standardized terminology (lab, medications,...)
ļ¼ Computer interpretable representations of guidelines/BPs
ļ¼ Environment to trial them at the point of care
ļ¼ Translate into routine services
Current Challenges Cont.ā¦
12
ļ± Architecture
ļ¼ Integrated into EHR
ļ¼ Context sharing and management
ļ± Implementation and Integration
ļ¼ Extraordinary variability in clinical practice patterns and workflow
ļ¼ Inopportune time in the workflow
ļ¼ Too late
ļ¼ Difficulty in developing, maintaining and integrating the clinical logic
Current Challenges
13
Develop Machine Learning
Model on ad-hoc Basis
ā¢ Development and operational
team are in silos
ā¢ Operationalization of developed
ML model is difficult
Utilize the pre-built Data-driven
CDS from EHR System
ā¢ CDS Tools are not provider
customized
ā¢ CDS Tools do not capture local
clinical process
ā¢ Custom data captured in Health
System may not be utilized
Current Scenario
14
Optimum Scenario
Easy To Develop and Deploy Clinical Data Science (CDS) App
Scalable
Heterogeneous Data Ingestion, Storage and Consumption
Secure
Automation
Modular
15
Proposed Solutions
16
What is Data Science Pipeline?
Mount Sinai Data Science (DS) Pipeline is end-to-end infrastructure
allowing:
ā¢ Consumption of a variety of historical and streaming data,
ā¢ Feature engineering and transformation,
ā¢ Building an CDS App, and
ā¢ finally operationalizing the CDS App
17
Computational Infrastructure
In
-house
ā¢
Mirth Connect
ā¢
MongoDB Cluster
ā¢
Apache Kafka
ā¢
Apache Spark
Cluster
ā¢
VMs
ā¢
MySQL DB
Cloud
ā¢
Microsoft Azure
18
Data Science Toolkit
Module
Packages
Descriptions
Examples
ETL Module
IO Utility
Functions related to read/write
from/to any source/destination
writeToHive()
Normalization
Data normalizations functions
mapGenericLabNameToLONIC()
Data Service
Receive various type of data
getVitals(), getPatientInfo()
Transformation
Performs various data
transformation
calculateAge()
getLabTimeSeries()
ML Module
Supervised
Learning
Wrapper over the different machine
learning algorithms
getCrossValidatedSVMModel()
getCrossValidatedLSTMModel()
Encoding
Functions to transform categorical
data into dummy variables
encodeCategoricalColumn()
Null Value
Imputation
Methods to impute numerical null
value
imputeNullValueByMean()
imputeNullValueByMode()
Feature
Engineering
Methods to create new features
calcualteBMI()
calculateLastVisitLengthOFStay()
Feature Selection
Methods for Multiple Features
Selection Techniques
getTopFeaturesByRecrusiveFeat
ureSelection()
Sampling
Different Sampling Techniques
conductSMOTESampling
()
conductOverSampling
()
19
System Architecture
20
ML Pipeline
21
Build DS App that is
Productionalizable
ā¢ Use variable that are available at
the prediction time
ā¢ Model Complexity Vs Operational
Cost
ā¢ Proper exceptional handling
Deployment Infrastructure
ā¢ Continuous Deployment Pipeline
ā¢ Dockerizing of a DS App
ā¢ Automated Monitoring, Debugging and
Alerts of DS App
ā¢ Continuous Monitoring of Data Quality
ā¢ Model Performance Dashboard
ā¢ 24/7 Support and Debugging
Operationalizing the Solution
22
Continuous Learning Engine
23
Practice Flow
ā¢ Ingesting stream of data from EHR, LAB, MED, ADT platforms -> outdated
data
ā¢ Real-time computation too late
ā¢ Real-time integration into point of care too late
ā¢ Using machine learning approaches to maintain clinical logic consistent and
patient specific
ā¢ Applying machine learning to optimize the golden time and minimize the
failure to rescue too late
ā¢ Streaming Data Service to extract and normalize data real-time data
collection/ data quality check/ data standardization for computation
ā¢ Product development cycle starts from frontline clinicians change in the
usersā mental mode / minimize irrelevancy
24
Incorporate ML Process into Current
Care Practice
25
Monitoring and Debugging
26
ML Pipeline Monitoring: Apps
MonitoringāApache Spark UI
27
ML Pipeline Monitoring: Apps
MonitoringāCustom UI
28
DS Infrastructure Monitoring Dashboard
ā¢ Nagios
ā¢ Mongo Ops Manager
29
ML Model Performance Evaluation
DashboardāCustom UI
30
USE CASE I: Early Warning System
for Patient Deterioration
Analysis of Main Hospital
31
Classical Approach
Limitations:
ļ§ BPA Alert--No Golden Time for Intervention
ļ§ Low Performance
ļ§ Low Accuracy (~ 70% )
ā¢ Median of the MEWS (~ 1). It suggests that MEWS fails in appraising the impact of the
presence of multiple co-morbidities
ļ§ Low Sensitivity (~ 60%)
Reference: https://www.rn.com/nursing-news/patient-deterioration-early-warning-signs/
32
Machine Learning Model To Predict
Patientās Deterioration After 16 hours
33
Comparing Classical MEWS with MEWS-
Plus
34
Use Case II: Early Detection of the
Risk of Severe Malnutrition in the
Inpatient Setting
Analysis of Main Hospital
34
35
Prediction Performance: Selected Model
Model
Test
Size
Malnutrition
Rate(%)
Sensitivity
Specificity
Accuracy
False Neg.
Rate(%)
False Pos.
Rate(%)
RF Classifier
2017 12.6 77.2 73.2 73.7 22.8 26.8
# of Trees
500
Max # of Bins
32
Max Depth of Trees
10
#
of Folds in CV 10
Feature Subset Strategy
1/3 featurs (394) = 131
ROC Curve: ROC Score=
0.82
36
Prediction Performance: Selected Model
vs. MUST
22.9
3.3
77.2
26.8
96.7
77.1
73.2
22.8
37
Challenges and Future Vision
37
38
Policy and Governance
ā¢ Financial incentives go frequently against data sharing for example in fee for
service business model
ā¢ Lack of Data governance initiatives in Health systems
ā¢ Paradigm shift in guidelines and best practices from empirical intervention to
proactive preventive intervention in acute care facilities
ā¢ Reinforce clinical documentation templates and flow sheets
39
Technical Challenges
ā¢ Multiple data sources and the lack of a unified data dictionary
across the EHR systems: The data science engine ingests data from
multiple data producers each having their own data representation as well as
custom extract, transform, and load logic.
ā¢ Data Quality: The majority of data were highly sparse and qualitative in
nature, which required considerable effort in terms of data preparation.
ā¢ Lack of Standardized Blue Print: Engineering/infrastructure teams are
yet to determine the optimal process
40
Future Plan and Vision
ā¢ Incorporate Outpatient Setting Data
ā¢ Incorporate Imaging, and Genomics Data
ā¢ Create Application Programming Interface(API) over Data Science Toolkit,
and publish API for usage within the Health Systems
ā¢ Move pipeline to cloud
ā¢ Provide DS App as a service for other providers nationally and
internationally
41
ā¢ All of this is possible thanks to a strong dedication to IT investments from
Mount Sinai Health System President & CEO Dr. Ken Davis & CIO Kumar
Chatani
ā¢ Project sponsored by the Mount Sinai Hospital President Dr. David Reich
Acknowledgement
